Mask then classify: multi-instance segmentation for surgical instruments.

PURPOSE The detection and segmentation of surgical instruments has been a vital step for many applications in minimally invasive surgical robotics. Previously, the problem was tackled from a semantic segmentation perspective, yet these methods fail to provide good segmentation maps of instrument types and do not contain any information on the instance affiliation of each pixel. We propose to overcome this limitation by using a novel instance segmentation method which first masks instruments and then classifies them into their respective type. METHODS We introduce a novel method for instance segmentation where a pixel-wise mask of each instance is found prior to classification. An encoder-decoder network is used to extract instrument instances, which are then separately classified using the features of the previous stages. Furthermore, we present a method to incorporate instrument priors from surgical robots. RESULTS Experiments are performed on the robotic instrument segmentation dataset of the 2017 endoscopic vision challenge. We perform a fourfold cross-validation and show an improvement of over 18% to the previous state-of-the-art. Furthermore, we perform an ablation study which highlights the importance of certain design choices and observe an increase of 10% over semantic segmentation methods. CONCLUSIONS We have presented a novel instance segmentation method for surgical instruments which outperforms previous semantic segmentation-based methods. Our method further provides a more informative output of instance level information, while retaining a precise segmentation mask. Finally, we have shown that robotic instrument priors can be used to further increase the performance

Anisotropic Quantum Spin Chains

We have studied two models for anisotropic quantum spin chains. (i) XY‐chain with a field in the plane: The magnetization of the ferromagnet behaves as h1/3 for small fields, in agreement with scaling laws. The antiferromagnet shows a critical field at which the ground state is a simple Néel state and which separates power law from exponential decay of spatial correlations. (ii) Anisotropic XY‐chain: The dynamic z‐component spin correlation function can be decomposed into a spin wave and a soliton contribution. The nature of quantum soliton excitations is studied and their form compared to soliton solutions of classical equations of motion

Learning how to robustly estimate camera pose in endoscopic videos.

PURPOSE Surgical scene understanding plays a critical role in the technology stack of tomorrow's intervention-assisting systems in endoscopic surgeries. For this, tracking the endoscope pose is a key component, but remains challenging due to illumination conditions, deforming tissues and the breathing motion of organs. METHOD We propose a solution for stereo endoscopes that estimates depth and optical flow to minimize two geometric losses for camera pose estimation. Most importantly, we introduce two learned adaptive per-pixel weight mappings that balance contributions according to the input image content. To do so, we train a Deep Declarative Network to take advantage of the expressiveness of deep learning and the robustness of a novel geometric-based optimization approach. We validate our approach on the publicly available SCARED dataset and introduce a new in vivo dataset, StereoMIS, which includes a wider spectrum of typically observed surgical settings. RESULTS Our method outperforms state-of-the-art methods on average and more importantly, in difficult scenarios where tissue deformations and breathing motion are visible. We observed that our proposed weight mappings attenuate the contribution of pixels on ambiguous regions of the images, such as deforming tissues. CONCLUSION We demonstrate the effectiveness of our solution to robustly estimate the camera pose in challenging endoscopic surgical scenes. Our contributions can be used to improve related tasks like simultaneous localization and mapping (SLAM) or 3D reconstruction, therefore advancing surgical scene understanding in minimally invasive surgery

2017 Robotic Instrument Segmentation Challenge

In mainstream computer vision and machine learning, public datasets such as ImageNet, COCO and KITTI have helped drive enormous improvements by enabling researchers to understand the strengths and limitations of different algorithms via performance comparison. However, this type of approach has had limited translation to problems in robotic assisted surgery as this field has never established the same level of common datasets and benchmarking methods. In 2015 a sub-challenge was introduced at the EndoVis workshop where a set of robotic images were provided with automatically generated annotations from robot forward kinematics. However, there were issues with this dataset due to the limited background variation, lack of complex motion and inaccuracies in the annotation. In this work we present the results of the 2017 challenge on robotic instrument segmentation which involved 10 teams participating in binary, parts and type based segmentation of articulated da Vinci robotic instruments

Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery

Intraoperative segmentation and tracking of minimally invasive instruments is a prerequisite for computer- and robotic-assisted surgery. Since additional hardware like tracking systems or the robot encoders are cumbersome and lack accuracy, surgical vision is evolving as promising techniques to segment and track the instruments using only the endoscopic images. However, what is missing so far are common image data sets for consistent evaluation and benchmarking of algorithms against each other. The paper presents a comparative validation study of different vision-based methods for instrument segmentation and tracking in the context of robotic as well as conventional laparoscopic surgery. The contribution of the paper is twofold: we introduce a comprehensive validation data set that was provided to the study participants and present the results of the comparative validation study. Based on the results of the validation study, we arrive at the conclusion that modern deep learning approaches outperform other methods in instrument segmentation tasks, but the results are still not perfect. Furthermore, we show that merging results from different methods actually significantly increases accuracy in comparison to the best stand-alone method. On the other hand, the results of the instrument tracking task show that this is still an open challenge, especially during challenging scenarios in conventional laparoscopic surgery

Heterogeneity of the Relative Benefits of TICI 2c/3 over TICI 2b50/2b67 : Are there Patients who are less Likely to Benefit?

PURPOSE Incomplete reperfusion after mechanical thrombectomy (MT) is associated with a poor outcome. Rescue therapy would potentially benefit some patients with an expanded treatment in cerebral ischemia score (eTICI) 2b50/2b67 reperfusion but also harbors increased risks. The relative benefits of eTICI 2c/3 over eTICI 2b50/67 in clinically important subpopulations were analyzed. METHODS Retrospective analysis of our institutional database for all patients with occlusion of the intracranial internal carotid artery (ICA) or the M1/M2 segment undergoing MT and final reperfusion of ≥eTICI 2b50 (903 patients). The heterogeneity in subgroups of different time metrics, age, National Institutes of Health Stroke Scale (NIHSS), number of retrieval attempts, Alberta Stroke Programme Early CT Score (ASPECTS) and site of occlusion using interaction terms (pi) was analyzed. RESULTS The presence of eTICI 2c/3 was associated with better outcomes in most subgroups. Time metrics showed no interaction of eTICI 2c/3 over eTICI 2b50/2b67 and clinical outcomes (onset to reperfusion pi = 0.77, puncture to reperfusion pi = 0.65, onset to puncture pi = 0.63). An eTICI 2c/3 had less consistent association with mRS ≤2 in older patients (>82 years, pi = 0.038) and patients with either lower NIHSS (≤9) or very high NIHSS (>19, pi = 0.01). Regarding occlusion sites, the beneficial effect of eTICI 2c/3 was absent for occlusions in the M2 segments (aOR 0.73, 95% confidence interval [CI] 0.33-1.59, pi = 0.018). CONCLUSION Beneficial effect of eTICI 2c/3 over eTICI 2b50/2b67 only decreased in older patients, M2-occlusions and patients with either low or very high NIHSS. Improving eTICI 2b50/2b67 to eTICI 2c/3 in those subgroups may be more often futile

Association of the 24‐Hour National Institutes of Health Stroke Scale After Mechanical Thrombectomy With Early and Long‐Term Survival

Background The National Institutes of Health Stroke Scale (NIHSS) obtained 24 hours after ischemic stroke is a good indicator for functional outcome and early mortality, but the correlation with long‐term survival is less clear. We analyzed the correlation of the NIHSS after 24 hours (24h NIHSS) and early clinical neurological development after mechanical thrombectomy with early and long‐term survival as well as its predictive power on survival. Methods We reviewed a prospective observational registry for all patients undergoing mechanical thrombectomy between January 2010 and December 2018. Vital status was extracted from the Swiss Population Registry. Adjusted hazard ratio (aHR) and crude hazard ratios were calculated using Cox regression. To assess predictive power of the 24h NIHSS, different Random Survival Forest models were evaluated. Results We included 957 patients (median follow‐up 1376 days). Patients with lower 24h NIHSS and major early neurological improvement had substantially better survival rates. We observed significantly higher aHR for death in patients with 24h NIHSS 12 to 15 (aHR, 1.78; 95% CI, 1.1–2.89), with 24h NIHSS 16 to 21 (aHR, 2.54, 95% CI, 1.59–4.06), and with 24h NIHSS >21 (aHR, 5.74; 95% CI, 3.47–9.5). The 24h NIHSS showed the best performance predicting mortality (receiver operating characteristic area under the curve at 3 months [0.85±0.034], at 1 year [0.82±0.029], at 2 years [0.82±0.031], and at 5 years [0.83±0.035]), followed by NIHSS change. Conclusions Patients with acute ischemic stroke achieving a low 24h NIHSS or major early neurological improvement after mechanical thrombectomy had markedly lower long‐term mortality. Furthermore, 24h NIHSS had the best predictive power for early and long‐term survival in our machine learning–based prediction

Long‐Term Outcome and Quality of Life in Patients With Stroke Presenting With Extensive Early Infarction

Background The benefit of mechanical thrombectomy in patients with low Alberta Stroke Program Early Computed Tomography Score (ASPECTS) for short‐term outcomes is debatable and long‐term outcomes remain unknown. This retrospective, monocentric cohort study aimed to assess the association between reperfusion grade and the long‐term functional outcome measured with modified Rankin scale as well as the long‐term health‐related quality of life recorded at the last follow‐up in patients according to baseline ASPECTS (0–5 versus 6–10). Methods Deceased patients were identified from the Swiss population register and follow‐up telephone interviews were conducted with all surviving patients with stroke treated with mechanical thrombectomy between January 1, 2010, and December 31, 2018. Favorable outcome was defined as modified Rankin scale 0 to 3; health‐related quality of life was assessed using the 3‐level version of the EuroQol 5‐dimensional questionnaire. The EuroQol 5‐dimension utility index was calculated for statistical analyses. The reperfusion grade was core laboratory adjudicated using the expanded treatment in cerebral ischemia score. Adjusted odds ratios for the association between the reperfusion grade assessed by expanded treatment in cerebral ischemia and outcomes were calculated from multivariable logistic regression. Results Of the 1114 patients with available long‐term follow‐up records (median follow‐up, 3.67 years), 997 were included in the final analysis. Respectively, patients with low ASPECTS more often had complaints regarding mobility (67.1% versus 42.1%, P<0.001), self‐care (53.4% versus 31.2%, P<0.001), and usual activities (65.8% versus 41.4%, P<0.001) than patients with high ASPECTS, whereas reported pain/discomfort (65.7% versus 69.9%, P=0.49) and anxiety/depression (71.2% versus 78.9%, P=0.17) did not differ. In patients with low ASPECTS, increasing reperfusion grade was associated with a higher likelihood of long‐term favorable functional outcome (adjusted odds ratio, 1.43; 95% CI, 1.09–1.88 [P=0.01]) and health‐related quality of life (adjusted linear correlation coefficient, 0.05; 95% CI, 0.02–0.08) despite early extensive infarction. Conclusion Despite low baseline ASPECTS, a higher reperfusion grade results in better functional outcomes and may improve health‐related quality of life in the long term

Multivariable Prediction Model for Futile Recanalization Therapies in Patients With Acute Ischemic Stroke.

BACKGROUND AND OBJECTIVES Very poor outcome despite intravenous thrombolysis (IVT) and mechanical thrombectomy (MT) occurs in about 1 of 4 patients with ischemic stroke and is associated with a high logistic and economic burden. We aimed to develop and validate a multivariable prognostic model to identify futile recanalization therapies (FRT) in patients undergoing those therapies. MATERIALS AND METHODS Patients from a prospectively collected observational registry of a single academic stroke center treated with MT and/or IVT were included. The dataset was split into a training (N=1808, 80%) and internal validation (N=453, 20%) cohort. We used gradient boosted decision tree machine-learning models after k-NN imputation of 32 variables available at admission to predict FRT defined as modified Rankin-Scale (mRS) 5-6 at 3 months. We report feature importance, ability for discrimination, calibration and decision curve analysis. RESULTS 2261 patients with a median (IQR) age 75 years (64-83), 46% female, median NIHSS 9 (4-17), 34% IVT alone, 41% MT alone, 25% bridging were included. Overall 539 (24%) had FRT, more often in MT alone (34%) as compared to IVT alone (11%). Feature importance identified clinical variables (stroke severity, age, active cancer, prestroke disability), laboratory values (glucose, CRP, creatinine), imaging biomarkers (white matter hyperintensities) and onset-to-admission time as the most important predictors. The final model was discriminatory for predicting 3-month FRT (AUC 0.87, 95% CI 0.87-0.88) and had good calibration (Brier 0.12, 0.11-0.12). Overall performance was moderate (F1-score 0.63 ± 0.004) and decision curve analyses suggested higher mean net benefit at lower thresholds of treatment (up to 0.8). CONCLUSIONS This FRT prediction model can help inform shared decision making and identify the most relevant features in the emergency setting. While it might be particularly useful in low resource healthcare settings, incorporation of further multifaceted variables is necessary to further increase the predictive performance